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Executive Summary 



According to the National Assessment of Educational Progress (NAEP), just over 70 
percent of students nationally arrive in high school with reading skills that are below “profi- 
cient” — defined as demonstrating competency over challenging subject matter. 1 Of these 
students, nearly half do not exhibit even partial mastery of the knowledge and skills that are 
fundamental to proficient work at grade level. 2 These limitations in literacy skills are a major 
source of course failure, high school dropout, and poor performance in postsecondary educa- 
tion. 3 While research is beginning to emerge about the special needs of striving adolescent 
readers, very little is known about effective interventions aimed at addressing these needs. 4 

To help fill this gap and to provide evidence-based guidance to practitioners, the U.S. 
Department of Education initiated the Enhanced Reading Opportunities (ERO) study — a 
demonstration and rigorous evaluation of supplemental literacy programs targeted to ninth- 
grade students whose reading skills are at least two years below grade level. 5 As part of this 
demonstration, 34 high schools from 10 school districts implemented one of two reading 
interventions: Reading Apprenticeship Academic Literacy (RAAL), designed by WestEd, and 
Xtreme Reading, designed by the University of Kansas Center for Research on Learning. These 
programs were implemented in the study schools for two school years. The U.S. Department of 
Education’s (ED) Office of Elementary and Secondary Education (OESE) 6 funded the imple- 
mentation of these programs, and its Institute of Education Sciences (IES) was responsible for 
oversight of the evaluation. MDRC — a nonprofit, nonpartisan education and social policy 
research organization — conducted the evaluation in partnership with the American Institutes 
for Research (AIR) and Survey Research Management (SRM). 

The goal of the reading interventions — which consist of a year-long course that re- 
places a ninth-grade elective class — is to help striving adolescent readers develop the strategies 
and routines used by proficient readers, thereby improving their reading skills and ultimately, 



Uhe NAEP is a nationally representative assessment of student achievement overseen by the National 
Center for Education Statistics (http://nces.ed.gov/nationsreportcard/). 

2 Lutkus, Rampey, and Donahue (2006) provide an analysis of NAEP reading results for urban school dis- 
tricts in the context of the national NAEP performance trends. 

3 Camevale (2001); Kamil (2003); Snow and Biancarosa (2003). 

4 Biancarosa and Snow (2004). 

5 The ERO study is known more formally as “An Evaluation of the Impact of Supplemental Literacy Inter- 
ventions in Freshman Academies.” 

6 The implementation was initially funded by the Office of Vocational and Adult Education (OVAE), but 
this role was later transferred to OESE. 
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their academic performance in high school. The first two reports for the study evaluated the 
programs’ impact on the two most proximal outcomes targeted by the interventions — students’ 
reading skills and their reading behaviors at the end of ninth grade. 7 This report — which is the 
final of three reports for this evaluation — examines the impact of the ERO programs on the 
more general outcomes that the programs hope to affect — students’ academic performance in 
high school (grade point average [GPA], credit accumulation, and state test scores) as well as 
students’ behavioral outcomes (attendance and disciplinary infractions). These academic and 
behavioral outcomes are examined during the year in which they were enrolled in the ERO 
programs (ninth grade), as well as the following school year (tenth grade for most students). 

Overall, the findings from these reports show that over the course of ninth grade, the 
ERO programs improved students’ reading comprehension skills and helped them perform 
better academically in their high school course work. However, these benefits did not persist in 
the following school year, when students were no longer receiving the supports provided by the 
ERO programs. The key findings from the study are the following: 8 

• The ERO programs improved students’ reading comprehension skills over the 
course of ninth grade. Across both cohorts of participating ninth-grade students, the 
ERO programs improved students’ reading comprehension scores by an effect size of 
0.09, 9 corresponding to an improvement from the twenty-third percentile to the 
twenty-fifth percentile nationally. However, 77 percent of students assigned to the 
ERO classes were still reading at two or more years below grade level at the end of 
ninth grade. 

• During the ninth grade, the ERO programs also had a positive impact on 
students’ academic performance in core subject areas. Students’ GPA in core 
subject areas 10 was 0.06 point higher (out of a maximum of 4 points) as a result of 
being assigned to the ERO program (effect size = 0.07). The programs also helped 
students earn 0.6 percentage point more of the core credits that they need to graduate 
(effect size = 0.06). In the subset of high schools located in states where standardized 
tests are administered in ninth grade, students also scored higher on their English lan- 



7 The first report presented implementation and impact findings for the first year of program operations 
(Kemple et al., 2008), while the second report focused on impacts in the second year of implementation (Corrin 
et al., 2008). Chapter 3 of this report provides a review of the implementation and impact findings from these 
two reports and also presents the average impact of the reading programs across both cohorts. 

8 The statistical significance of all impact estimates in this report is evaluated at the 5 percent level. 

9 In this report, effect sizes are expressed as a proportion of the overall variability (standard deviation) in 
the outcome measure among students who were not assigned to the reading program (non-ERO group). 

10 The four core subject areas are English language arts, social studies, science, and mathematics. 
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guage arts and mathematics tests as a result of having been assigned to the ERO pro- 
gram; the estimated effect size of these impacts are 0.11 and 0.07, respectively. 

• However, in the school year following students’ participation in the ERO 
programs, the programs no longer had an impact on academic performance. 

Estimated impacts on students’ GPA in core subject areas, credit accumulation, and 
standardized state test scores are not statistically significant in the school year follow- 
ing program participation (tenth grade for most students). 

• The ERO programs did not increase students’ vocabulary scores, nor did the 
programs affect students’ reading behaviors or their school behaviors. The pro- 
grams did not have a statistically significant impact on students’ vocabulary scores at 
the end of ninth grade. Nor did the programs have a statistically significant effect on 
how often students read school-related or non-school-related texts, or on how often 
students use the reading strategies taught by the two programs. Similarly, impacts on 
student attendance and suspensions were not statistically significant, in either the 
program year or the following school year. 

The first two study reports also examined how well the ERO programs were imple- 
mented in the study schools, as well as the extent to which the experience of students in the 
ERO programs compared with the literacy support received by students not selected for the 
programs. A key finding from these reports is that, by the end of the second year of program 
operation, implementation of the reading interventions, as rated through classroom observation, 
was well aligned with the respective program models. In addition, schools were able to offer the 
programs for the entire school year. In both implementation years, students in the ERO class 
received a greater amount of literacy support than they would have received had they not been 
assigned to the program. 



Overview of the ERO Study 

The ERO study is both a demonstration and a rigorous evaluation of two established 
supplemental literacy interventions that are targeted to ninth-grade students whose reading skills 
are two or more years below grade level as they enter high school. The purpose of the study is 
to evaluate these interventions’ impact on students’ reading comprehension skills and their 
academic performance as they move through high school. See Box ES.l for a brief overview of 
the components of the ERO study. 
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Box ES.1 



Overview of the Study 

Interventions: Reading Apprenticeship Academic Literacy (RAAL) and Xtreme Reading — 
supplemental literacy programs designed as full-year courses to replace a ninth-grade elective 
class. The programs were selected through a competitive applications process based on ratings by 
an expert panel. 

Study sample: Two cohorts of ninth-grade students from 34 high schools and 10 school districts 
(2,916 students in Cohort 1 and 2,679 students in Cohort 2). Districts and schools were selected 
by ED’s Office of Vocational and Adult Education through a special Small Learning Communi- 
ties Grant competition. Students were selected based on reading comprehension test scores that 
were between two and five years below grade level. 

Research design: Within each district, high schools were randomly assigned to use either the 
RAAL program or the Xtreme Reading program during two school years (2005-2006 and 2006- 
2007). Within each high school, students were randomly assigned to enroll in the ERO class 
(ERO group) or to remain in a regularly scheduled elective class (non-ERO group). Because 
students were randomly assigned to the ERO program, the impact of the programs can be esti- 
mated by comparing the outcomes of students in the ERO and the non-ERO group. Impact 
estimates are regression-adjusted for the blocking of random assignment as well as random 
baseline differences between the ERO and non-ERO group. 

Data collection: Classroom observations in the first and second semester of the school year were 
used to measure implementation fidelity. A reading comprehension test — the Group Reading 
Assessment and Diagnostic Examination (GRADE) — and a survey were administered to stu- 
dents in the spring of eighth grade or at the start of ninth grade prior to random assignment, and 
again at the end of ninth grade. School records data were collected for students’ ninth-grade and 
tenth-grade year (and for Cohort 1 ’s eleventh-grade year); these data include course transcripts, 
state test scores, attendance, and disciplinary outcomes. 

Outcomes: Reading comprehension test scores, vocabulary test scores, and self-reported reading 
behaviors at the end of ninth grade; grade point average, credit accumulation, state test scores, 
school attendance and suspensions during ninth grade and in the following school year. 



A Demonstration of Supplemental Literacy Interventions 

The two reading programs selected for the ERO study, RAAL and Xtreme Reading, 
were selected for the study from a pool of 17 applicants by a national panel of experts on 
adolescent literacy. 11 The applicants had responded to a public request for supplemental reading 



u For an overview of research related to RAAL, see Schoenbach, Greenleaf, Cziko, and Hurwitz (1999). 
For an overview of research related to Xtreme Reading and the Strategic Instruction Model, see Schumaker 
and De shier (2003, 2004). 
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programs that would substantially improve students’ reading skills. The programs share 
common core goals and instructional strategies, and can therefore be considered part of the 
same broad class of literacy intervention for struggling adolescent readers. The short-term goal 
of the programs is to help ninth-grade students adopt the strategies and routines used by profi- 
cient readers, to improve their comprehension skills, and to motivate them to read more and to 
enjoy reading. To do so, each program supports instruction in the following areas: (1) student 
motivation and engagement; (2) reading fluency, or the ability to read quickly, accurately, and 
with appropriate expression; (3) vocabulary, or word knowledge; (4) comprehension, or making 
meaning from text; (5) phonics and phonemic awareness (for students who could still benefit 
from instruction in these areas); and (6) writing. 

By addressing the reading needs of students, the programs also aim to improve stu- 
dents’ performance in their high school courses and on standardized assessments, thereby 
helping them meet the milestones required for grade promotion and graduation. The programs 
seek to improve these longer-term outcomes by targeting not only students’ reading skills but 
also their content literacy — by identifying, modeling, and explaining context-specific strategies 
that are most applicable in English language arts, science, and social studies texts (for example, 
differences in text structures). Finally, both interventions have components that promote 
positive behavioral nonns, which in turn may improve students’ attendance rates and reduce 
their disciplinary infractions. 

Experienced, full-time English language arts or social studies teachers volunteered to 
teach the ERO class and were approved by ED, the districts, and the schools to teach the 
programs for a period of two years. During each year of the project, the programs’ developers 
provided three types of training and technical assistance to the ERO teacher from each school: 
(1) a summer training institute before the start of the school year (five days in the first year of 
the study and three days in the second year); (2) booster training sessions; and (3) coaching 
visits during the school year (a minimum of two one-day visits during the first year and three 
two-day visits during the second year). 

Each ERO teacher (one per school) was responsible for teaching four sections of the 
ERO class. Each section accommodated between 1 0 and 1 5 students. Classes were designed to 
meet for a minimum of 225 minutes per week and were scheduled as a 45 -minute class every 
day or as a 75- to 90-minute class that met every other day during the school year. As noted 
earlier, the classes are supplemental, in that they replace a ninth-grade elective class, rather than 
a core academic class, and in that they are offered in addition to students’ regular English 
language arts classes. The average annual cost of the programs, as implemented, was $1,931 per 
student. Salary expenditures represent the largest portion of this cost (72 percent). An additional 
13.4 percent of the per-student cost was spent on training resources, while 5.9 percent paid for 
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travel to and from training activities. The remaining 8.5 percent covered equipment/supplies, 
other direct costs, and indirect costs. 12 

A Rigorous Impact Evaluation 

The supplemental reading programs were implemented in 34 high schools from 10 
school districts across the country. The districts were selected through a special grant competi- 
tion organized by the U.S. Department of Education’s Office of Vocational and Adult Educa- 
tion (OVAE). As an extension of the Smaller Learning Communities (SLCs) grant program, 
this competition sought to provide funding for the implementation of two supplemental ninth - 
grade literacy programs in selected high schools and to sustain and enhance existing SLCs in 
these high schools. 

The ERO evaluation uses a two-level random assignment research design. First, within 
each district, high schools were randomly assigned to use one of the two supplemental literacy 
programs: 17 high schools were assigned to use RAAL, and 17 schools were selected to use 
Xtreme Reading. Each school implemented the same program in both the 2005-2006 and 2006- 
2007 school years. In the second stage of the study design, eligible students within each of the 
participating high schools were randomly assigned either to enroll in the ERO class (ERO 
group) or to take one of their school’s regularly offered elective classes (non-ERO group). 

Across both years of implementation, the participating high schools identified 5,595 
ninth-grade students reading two to five years below grade level (an average of 82 students per 
school per cohort). Fifty-seven percent of these students were randomly assigned to the ERO 
group and 43 percent randomly assigned to the non-ERO group. Random assignment resulted in 
two groups of students that were similar with respect to their background characteristics and 
prior achievement at the start of the study. 

The ERO evaluation uses a variety of data sources to measure impacts on student out- 
comes as well as the nature and quality of program implementation. To learn about the fidelity 
with which the programs were implemented, the study conducted observations of the supple- 



12 Program costs were calculated by identifying the key program inputs and resources (that is, personnel 
salaries and fringe benefits, training and travel costs, equipment and supplies, other direct costs, and indirect 
costs), and then calculating program costs based on the unit cost and quantities of these resources. Costs per 
student were then obtained by dividing the total cost by the number of students enrolled in the ERO programs 
in the study schools. Cost data were obtained from documents prepared by districts as part of their application 
for the Smaller Learning Community (SLC) grant, developer budget documents, teacher salary step schedules 
obtained from district Web sites, and the National Center for Education Statistics (NCES) Common Core of 
Data (CCD) Fiscal Survey (F-33 data). 
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mental literacy classes during the first and second semester of the school year. To measure the 
amount of literacy instruction received by students — as well as the contrast between the 
nature and quantity of the literacy services received by students in the ERO and non-ERO 
groups — the study team used a combination of attendance records from ERO teachers and a 
student survey. To measure program effectiveness, data were collected on four types of student 
outcomes: reading achievement (reading comprehension and vocabulary scores from the 
Group Reading Assessment and Diagnostic Examination, also known as the GRADE), 13 
reading behaviors (frequency of reading and use of reading comprehension strategies from a 
student survey), academic perfonnance in core subject areas (GPA, credits earned, and state 
assessment scores from school records), and school behaviors (attendance and disciplinary data 
from school records). 



Implementation Findings 

The ERO study examined three aspects of program implementation that may influence 
whether and by how much the reading interventions can improve student outcomes: (1) the 
extent to which the study schools were implementing the ERO programs as specified by the 
program developers (implementation fidelity), (2) the amount of ERO instruction received by 
students (dosage), and (3) whether the literacy services received by students in the ERO group 
differed in amount and type from the services received by students in the non-ERO group 
(service contrast). These implementation findings are discussed in the first two study reports 
and summarized below. 

• By the end of the second year, most participating high schools’ implementation of the 
ERO programs was well aligned with the program models. 

During each year of the project, observers from the study team used a structured class- 
room observation protocol to examine whether the study schools were implementing the ERO 
programs as specified by the program developers. 14 The overall implementation of the ERO 
program in a given school was classified as “well aligned,” “moderately aligned,” or “poorly 



13 The GRADE is a norm-referenced, research-based reading assessment that is used widely to measure 
performance and track the growth of an individual student and groups of students. The average score on the 
reading comprehension or vocabulary subtests is 100 for a nationally representative group of students at the 
end of their ninth-grade year. The national standard deviation of scores for both tests is 15. For more informa- 
tion, see American Guidance Service (2001a, 2001b). 

14 The analysis of implementation fidelity is based on three field research visits to each of the 34 high 
schools — one during the second semester of the first implementation year, and one in each of the first and 
second semesters of the second implementation year. 
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aligned,” based on observers’ ratings of how reflective the ERO classroom’s learning environ- 
ment (classroom climate) and comprehension instruction (the teacher’s use of ERO instructional 
strategies) were of the behaviors and activities specified by the developers. At the spring site 
visit in the second year of the study, the ERO courses at 26 of the 34 schools (76 percent of 
schools) were categorized as being well aligned with the program models. 

• In the second year of the study, schools operated the ERO programs for the entire 
school year, and student participation was close to what was intended. 

To measure the amount of ERO instruction received by students (dosage), the study 
team collected data on the duration of the ERO classes as well as the frequency with which 
students attended the ERO classes, from ERO teachers’ attendance records. On average, 
students in the ERO group attended 79 percent of scheduled ERO classes, and they received 98 
hours of ERO instruction during the school year, or 1 1 hours per month, which is the same 
monthly average as in the first year of implementation. The programs were designed to meet for 
a minimum of 225 minutes per week (15 hours per month), and the schools were able to offer 
the ERO classes an average of 14 hours per month. Thus, the 1 1 hours per month of ERO 
instruction received by students represents 79 percent of the amount offered and 73 percent of 
the amount intended. 

• In both implementation years, students in the ERO group reported a higher frequency 
of participation in supplemental literacy services than students in the non-ERO 
group. 

For a program to have an impact on outcomes, it needs to provide services that differ 
(for example, in quality, nature, frequency) from the services students would have otherwise 
received; this difference is known as the “service contrast.” The ERO study team collected data 
to better understand what types of literacy services were received by the non-ERO students and 
the extent to which such supports may have reduced the service contrast between the ERO and 
non-ERO group. A student survey (administered at the end of ninth grade) was used to measure 
ERO and non-ERO students’ participation in different types of supplementary literacy support 
activities (including the ERO class). 15 These data confirm that, as expected, students in the ERO 
group attended a greater number of school-based literacy classes annually than students in the 
non-ERO group. On average, the ERO students reported attending 52 more literacy class 
sessions than the non-ERO students in the first year of implementation and 58 more sessions in 
the second year. 



15 Although students in the ERO group were not specifically instructed to include ERO classes in their 
survey response, it is highly likely that their responses include this class. 
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Impact Findings 

The previous two study reports — which focused on the effect of the programs on stu- 
dents’ reading outcomes in ninth grade — showed that the ERO programs improved students’ 
reading comprehension scores. Students’ reading achievement at the end of ninth grade was 
measured using the reading comprehension and vocabulary subtests in the GRADE. 

• The ERO programs improved students’ reading comprehension scores over the 
course of ninth grade. 

Figure ES.l shows the impact of the ERO programs on students’ GRADE reading 
comprehension test scores in standard score units at the end of ninth grade. 16 The impact on 
students’ reading comprehension scores was 0.9 standard score point (effect size = 0.09). 17 As 
seen in the figure, students in the ERO group started their ninth-grade year with an average 
standard score of 85.3, which corresponds, approximately, to a grade equivalent of 5.0 (the start 
of fifth grade) and a reading level at the fifteenth percentile for ninth-grade students nationally. 
In the spring of ninth grade after enrolling in the program, the ERO group’s average score was 
90.1 points (twenty-fifth percentile nationally), which means that these students’ reading scores 
increased by 4.9 points over the course of ninth grade. In contrast, had these students not been 
assigned to the ERO class, their reading scores would have increased by 4.0 standard score 
points during the year (the estimated growth of the non-ERO group), to 89.2 points or the 
twenty-third percentile nationally. The impact of the ERO programs is the difference between 
the growth in test scores of the two groups (0.9 standard score point) and represents an 23 
percent improvement over and above the growth that the ERO group would have experienced if 
they had not had the opportunity to attend the ERO classes (4.0 points). 

Despite this impact, ERO students’ reading skills were still below grade level at the end 
of the program. As shown by the solid line at the top of Figure ES.l, students with a score of 
100 points on the GRADE reading comprehension test at the end of ninth grade are considered 
to be reading at grade level. In contrast, students in the ERO group had an average standard 
score of 90.1 at the end of ninth grade, which means that ERO students were nearly 10 points 
below the national average, or almost four years below grade level. More concretely, 77 percent 
of students in the ERO group would still be eligible for the ERO programs were these programs 



16 Impact estimates are regression-adjusted using ordinary least squares, controlling for the blocking of 
random assignment by school and cohort and for random differences between the ERO and non-ERO groups 
in their baseline characteristics and prior achievement. 

17 The impact on students’ reading comprehension scores does not differ by a statistically significant 
amount between the two implementation years (effect size = 0.09 for students in the first cohort and 0.08 for 
students in the second cohort). 
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The Enhanced Reading Opportunities Study 
Figure ES.l 

Impacts on Reading Achievement, 
GRADE Respondent Sample 
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SOURCE: MDRC calculations from the Enhanced Reading Opportunities GRADE assessment, administered 
at the end of ninth grade (spring 2006 for Cohort 1 and spring 2007 for Cohort 2). 

NOTES: The estimated impacts are regression-adjusted using ordinary least squares, controlling for the 
blocking of random assignment by school and cohort, as well as random baseline differences between the 
ERO and non-ERO groups in terms of the following variables: GRADE reading comprehension test score at 
baseline and whether a student was overage for grade at random assignment. The ERO group growth at 
follow-up is calculated as the difference between the unadjusted ERO group mean at baseline and the 
unadjusted ERO group mean at follow-up. The expected ERO group growth at follow-up is the difference 
between the actual ERO group growth and the impact. 

A two-tailed t-test was applied to the impact estimate. The statistical significance is indicated (*) when 
the p-value is less than or equal to 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“The national average for standard scores is 100, and its standard deviation is 15. The grade equivalent 
and percentile are those associated with the average standard score as indicated in the GRADE Teacher's 
Scoring and Interpretive Manual (Level H, Grade 9, Spring Testing, Form B). No statistical tests or 
arithmetic operations were performed on these reference points. 
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again made available to them (because they scored two or more years below grade level at the 
end of their ninth-grade year). 

This report presents new findings related to the ERO programs’ impact on students’ 
academic perfomiance and their behavioral outcomes, in ninth grade (the “program year”) and 
in the subsequent school year (tenth grade for most students). School records provided by the 
study districts included information on students’ GPA in core subject areas (English language 
arts, social studies, and science), the number of credits earned by students in these subject areas, 
and students’ scores on the tests mandated by their state. 

• The ERO programs had a positive impact on students’ GPA and credits earned while 
students were in the program (ninth grade), but these impacts did not persist into the 
following school year. 

As shown in the top panel of Figure ES.2, students in the ERO group had a GPA of 
1 .60 points during their ninth-grade year (out of a maximum of 4 points), while students in the 
non-ERO group had a GPA of 1.53 points, which means that both groups of students had a D 
average during the program year. However, the GPA of students in the ERO group was statisti- 
cally higher than that of students in the non-ERO group (a difference of 0.06 point; effect size = 
0.07). This improvement is such that ERO students were 13 percent closer to achieving C 
average (2.0 points), which is an important milestone associated with a higher likelihood of 
graduating from high school. 18 ’ 19 

As shown in the top panel of Figure ES.3, the ERO programs also helped students ac- 
cumulate a greater number of credits in core subject areas. By the end of ninth grade, students in 
the ERO group had earned 2.99 credits (or 21.4 percent of the core credits that they need to 
graduate), while students in the non-ERO group had earned 2.91 credits (20.9 percent of the 
core credits needed to graduate). To put these findings into perspective, consider that at the end 
of ninth grade, students should have earned 25 percent of the core credits required for gradua- 
tion in order to be “on track” to graduate. Therefore, by the end of the program year, neither the 
ERO group nor the non-ERO group was “on track” to graduate in four years. However, students 
in the ERO group had earned a greater percentage of required core credits than students in the 
non-ERO group (a difference of 0.6 percentage point; effect size = 0.06). This improvement is 



18 Based on Allensworth and Easton (2007), graduation rates in the Chicago Public Schools are 36 percentage 
points higher for students with a C average (2.0) than for students with a D average (1.0). A grade of C is also 
used by a majority of the study schools to determine “average” or “satisfactory” achievement in a given course. 

19 The value of 13 percent was calculated by dividing the impact (0.06 points) by the amount by which 
students in the ERO group are below a C average (0.47 points). 
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The Enhanced Reading Opportunities Study 
Figure ES.2 

Impacts on Grade Point Average (GPA), 
School Records Sample 




GPA in Core Subject Areas (corresponding letter grade) 



SOURCE: MDRC calculations from school records data. 

NOTES: The program year is the year in which students were enrolled in an ERO class; it corresponds to the 
2005-2006 school year for Cohort 1 and the 2006-2007 school year for Cohort 2. The follow-up year 
corresponds to the 2006-2007 school year for Cohort 1 and the 2007-2008 school year for Cohort 2. 

The estimated impacts are regression-adjusted using ordinary least squares, controlling for the blocking of 
random assignment by school and cohort, as well as random baseline differences between the ERO and non- 
ERO groups in terms of the following variables: GRADE reading comprehension test score at baseline, whether 
a student was overage for grade at random assignment, a student's score on the standardized reading and math 
assessments (in standardized units) administered by the school district in the year prior to ERO participation, 
and a baseline measurement of the outcome variable in the school year prior to ERO participation. The ERO 
group values are the unadjusted mean for the students randomly assigned to the ERO programs. The “Non-ERO 
Group” values are the regression-adjusted means for students randomly assigned to the non-ERO group, using 
the observed mean covariate values for the ERO group as the basis for the adjustment. 

A two-tailed t-test was applied to the impact estimate. The statistical significance is indicated (*) when the p- 
value is less than or equal to 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

GPA in core subject areas is based on a 4-point scale: A+/A/A- = 4.0; B+/B/B- = 3.0; C+/C/C- = 2.0; 
D+/D/D- = 1.0; F = 0.0. 
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The Enhanced Reading Opportunities Study 
Figure ES.3 



Impacts on Credit Accumulation 

(Credits Earned as a Percentage of Credits Required for Graduation), 

School Records Sample 




Credits Earned in Core Subject Areas (cumulative %) 



SOURCE: MDRC calculations from school records data. 

NOTES: The program year is the year in which students were enrolled in an ERO class; it corresponds to the 2005- 
2006 school year for Cohort 1 and the 2006-2007 school year for Cohort 2. The follow-up year corresponds to the 
2006-2007 school year for Cohort 1 and the 2007-2008 school year for Cohort 2. 

The estimated impacts are regression-adjusted using ordinary least squares, controlling for the blocking of 
random assignment by school and cohort, as well as random baseline differences between the ERO and non-ERO 
groups in terns of the following variables: GRADE reading comprehension test score at baseline, whether a 
student was overage for grade at random assignment, a student's score on the standardized reading and math 
assessments (in standardized units) administered by the school district in the year prior to ERO participation, and a 
baseline measurement of the outcome variable in the school year prior to ERO participation. The ERO group value 
is the unadjusted mean for the students randomly assigned to the ERO programs. The “Non-ERO Group” values 
are the regression-adjusted means for students randomly assigned to the non-ERO group, using the observed mean 
covariate values for the ERO group as the basis for the adjustment. 

A two-tailed t-test was applied to the impact estimate. The statistical significance is indicated (*) when the p- 
value is less than or equal to 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The cumulative number of credits earned is scaled as a percentage of the total number of core credits required 
for graduation in a student's district. 
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statistically significant, and its magnitude is such that students in the ERO group were 15 
percent closer to being “on-track” to graduate as a result of the ERO program. 20,21 

The ERO programs did not affect students’ GPA or credits earned in core classes in the 
school year following their participation in the ERO programs. The lower set of bars in Figure 
ES.2 shows the average GPA in core courses for ERO students and non-ERO students during 
the follow-up year (1.59 and 1.54 points, respectively). 22 The estimated impact of 0.04 point is 
not statistically significant. Nor did the ERO programs have a statistically significant impact on 
credits earned by the end of the following school year (lower set of bars in Figure ES.3). At the 
end of the follow-up year, on average, non-ERO students had accumulated 43.8 percent of the 
credits they need to graduate, while ERO students had accumulated 44.4 percent on average. 23 
The estimated impact of 0.5 percentage point is not statistically significant. Therefore, it cannot 
be concluded that the programs had improved students’ GPA or credit accumulation by the end 
of the follow-up year. 

• The ERO programs had a positive impact on students’ perfomiance on state 
tests in English language arts and mathematics in ninth grade. Impacts on 
state test scores in the following school year are not statistically significant. 

Table ES.l shows findings for the estimated impact of the ERO programs on students’ 
scores on the tests mandated by their state. In the subset of schools where state tests are ad- 
ministered in the ninth grade, the programs had a positive impact on students’ performance on 
state tests in English language arts (effect size = 0.11) and mathematics (effect size = 0.07). 
However, impacts on state test scores in the follow-up year were not statistically significant. 

• The ERO programs did not have a statistically significant impact on students’ 
reading vocabulary scores or on their reading behaviors, nor did it affect their 
school behaviors. 



20 The value of 15 percent was calculated by dividing the impact (0.6 percentage point) by the percentage 
of credits needed by the ERO group to attain 25 percent of their core course credits (4. 1 percentage points). 

21 Viewed otherwise, the ERO programs had an estimated impact of 0.08 on the number of core credits 
earned by students (the ERO group earned 2.99 credits on average in the program year, while the non-ERO 
group earned 2.91 credits). One credit represents a full-year course, so students in the ERO group earned an 
additional 8 percent of a full-year course credit relative to the non-ERO group. 

22 Note that the GPA measure is not cumulative and includes only students’ grades in core courses during 
the given school year. 

23 The measure of credit accumulation is defined cumulatively, in order to capture a student’s progress 
toward graduation. Thus, credit accumulation at the end of the follow-up year includes credits earned during 
the program year and the follow-up year. 
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The Enhanced Reading Opportunities Study 
Table ES.l 



Impacts on State Test Scores (Standardized), 
School Records Sample 



Outcome 


Number of 

Students ERO Group 


Non-ERO 

Group 


Estimated 
Impact 
Effect Size 3 


P- Value for 
Estimated 
Impact 


All schools 

Program year 


English language arts (ELA) 


2,244 


0.11 


0.01 


0.11* 


0.003 


Social studies 


952 


0.07 


0.01 


0.06 


0.343 


Science 


2,348 


0.07 


0.01 


0.06 


0.166 


Math 


2,668 


0.08 


0.01 


0.07* 


0.045 


Follow-up year 


English language arts (ELA) 


2,408 


0.00 


-0.01 


0.01 


0.705 


Social studies 


2,237 


-0.04 


0.01 


-0.04 


0.262 


Science 


2,661 


0.00 


0.02 


-0.02 


0.649 


Math 


2,537 


-0.03 


0.02 


-0.04 


0.265 



SOURCE: MDRC calculations from school records data. 



NOTES: The program year is the year in which students were enrolled in an ERO class; it corresponds to 
the 2005-2006 school year for Cohort 1 and the 2006-2007 school year for Cohort 2. The follow-up year 
corresponds to the 2006-2007 school year for Cohort 1 and the 2007-2008 school year for Cohort 2. 

The estimated impacts are regression-adjusted using ordinary least squares, controlling for the 
blocking of random assignment by school and cohort, as well as random baseline differences between 
the ERO and non-ERO groups in terms of the following variables: GRADE reading comprehension test 
score at baseline, whether a student was overage for grade at random assignment, a student's score on the 
standardized reading and math assessments (in standardized units) administered by their school district in 
the year prior to ERO participation, and a baseline measurement of the outcome variable in the school 
year prior to ERO participation. The ERO group value is the unadjusted mean for the students randomly 
assigned to the ERO programs. The “Non-ERO Group” values in the next column are the regression- 
adjusted means for students randomly assigned to the non-ERO group, using the observed mean 
covariate values for the ERO group as the basis for the adjustment. The estimated impact effect size is 
calculated as a proportion of the standard deviation of the outcome for the non-ERO group during the 
relevant year (all schools). 

A two-tailed t-test was applied to the impact estimate. The statistical significance is indicated (*) 
when the p-value is less than or equal to 5 percent. 

The numbers of students reported in this table are for students in the school records sample who have 
state test scores for a given subject area in the relevant year. A student may have taken more than one 
test in a given subject area in more than one year. If a student wrote a specific test more than once, only 
his or her first score is used. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“State test scores are standardized by district, follow-up year, and cohort, using the means and 
standard deviation of the non-ERO group. 



ES-15 




The previous two study reports showed that the ERO programs did not have a statisti- 
cally significant impact on students’ scores on the GRADE vocabulary subtest at the end of 
ninth grade. Nor did the programs have a statistically significant effect on the frequency with 
which students read inside or outside of school, or on the extent to which they use the different 
kinds of reading strategies taught by the ERO programs, based on a student survey administered 
at the end of ninth grade. The new findings in this report show that, similarly, the programs did 
not have a statistically significant effect on students’ attendance rate, nor did they affect whether 
students were suspended, in either ninth grade or the following school year. Information on 
these student behavioral outcomes was available from records provided by the study districts. 

• There is no conclusive evidence that the programs were more effective for one 
subgroup of students than another. 

In this report and in prior reports, the impact of the reading programs on reading and 
high school outcomes was estimated for subgroups of students defined by their baseline reading 
achievement level, whether or not they were overage for grade at the start of ninth grade, and 
whether or not a language other than English is spoken in their home. Based on these analyses, 
it cannot be concluded that the ERO programs’ impacts on students in a given subgroup were 
different from their impacts on students in the other subgroup(s) in that category. 

• Each of the two ERO programs had positive impacts for students during the program 
year, but the statistical certainty (significance) of the impacts varied by outcome. 

Because the primary goal of the ERO study is to estimate the impact of full-year sup- 
plemental reading programs, the findings in this report focus on the combined impact of the two 
reading programs together (RAAL, Xtreme Reading). However, in order to contextualize the 
overall impact findings, program-specific impacts are also examined in this report. Impact 
findings for each of the two reading programs separately tell a similar story to the pooled 
findings for both programs together. As shown in Table ES.2, both programs had a positive 
effect in ninth grade on GPA (statistically significant) and credit accumulation (not statistically 
significant). For both programs, these two impacts were similar in magnitude to the overall 
impacts for both programs pooled together. Neither of the two programs had a positive impact 
on academic performance in the follow-up year. For the GRADE reading comprehension test 
scores in ninth grade pooled across cohorts, the impact of RAAL was 1 .2 standard score points 
(effect size = 0.12) and statistically significant, and the impact of Xtreme Reading was 0.6 
standard score point (effect size = 0.05) and not statistically significant. The difference in the 
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The Enhanced Reading Opportunities Study 
Table ES.2 



Impacts on Grade Point Average (GPA) and Credit Accumulation in Core Subject Areas 
(Credits Earned as a Percentage of Credits Required for Graduation), 

School Records Sample 











Estimated 


P-Value for 






Non-ERO 


Estimated 


Impact 


Estimated 


Outcome 


ERO Group 


Group 


Impact 


Effect Size 


Impact 


All schools 












Program year 












GPA in core subject areas 


1.60 


1.53 


0.06 * 


0.07 * 


0.002 


Credits earned in core subject areas (%) 


21.4 


20.9 


0.6 * 


0.06 * 


0.017 


Sample size 


2,937 


2,213 








Follow-up year 












GPA in core subject areas 
Cumulative credits earned in 


1.59 


1.54 


0.04 


0.05 


0.061 


core subject areas (%) 


44.4 


43.8 


0.5 


0.03 


0.212 


Sample size 


2,542 


1,894 









Reading Apprenticeship schools 

Program year 



GPA in core subject areas 


1.54 


1.47 


0.07 * 


0.07 * 


0.019 


Credits earned in core subject areas (%) 


20.5 


19.9 


0.6 


0.06 


0.072 


Sample size 


1,468 


1,095 








Follow-up year 












GPA in core subject areas 


1.55 


1.52 


0.03 


0.03 


0.392 


Cumulative credits earned in 












core subject areas (%) 


42.6 


42.1 


0.4 


0.03 


0.486 


Sample size 


1,285 


927 









Xtreme Reading schools 

Program year 



GPA in core subject areas 


1.66 


1.60 


0.06 * 


0.06 * 


0.038 


Credits earned in core subject areas (%) 


22.3 


21.8 


0.5 


0.05 


0.138 


Sample size 


1,469 


1,118 








Follow-up year 












GPA in core subject areas 
Cumulative credits earned in 


1.62 


1.56 


0.06 


0.06 


0.068 


core subject areas (%) 


46.2 


45.5 


0.7 


0.04 


0.266 


Sample size 


1,257 


967 









(continued) 
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Table ES.2 (continued) 

SOURCE: MDRC calculations from school records data. 

NOTES: The program year is the year in which students were enrolled in an ERO class; it corresponds 
to the 2005-2006 school year for Cohort 1 and the 2006-2007 school year for Cohort 2. The follow-up 
year corresponds to the 2006-2007 school year for Cohort 1 and the 2007-2008 school year for Cohort 
2 . 

The estimated impacts are regression-adjusted using ordinary least squares, controlling for the 
blocking of random assignment by school and cohort, as well as random baseline differences between 
the ERO and non-ERO groups in terms of the following variables: GRADE reading comprehension 
test score at baseline, whether a student was overage for grade at random assignment, a student's score 
on the standardized reading and math assessments (in standardized units) administered by their school 
district in the year prior to ERO participation, and a baseline measurement of the outcome variable in 
the school year prior to ERO participation. The ERO group value is the unadjusted mean for the 
students randomly assigned to the ERO programs. The “Non-ERO Group” values in the next column 
are the regression-adjusted means for students randomly assigned to the non-ERO group, using the 
observed mean covariate values for the ERO group as the basis for the adjustment. The estimated 
impact effect size is calculated as a proportion of the standard deviation of the outcome for the non- 
ERO group during the relevant year (all schools). 

A two-tailed t-test was applied to the impact estimate. The statistical significance is indicated (*) 
when the p-value is less than or equal to 5 percent. 

The sample sizes reported in this table are for students with school records data in the relevant year. 
Because students may not have earned grades in all core subject areas in a given year, sample sizes 
differ for impacts in the specific core subject areas. 

Rounding may cause slight discrepancies in calculating sums and differences. 

GPA in core subject areas is based on a 4-point scale: A+/A/A- = 4.0; B+/B/B- = 3.0; C+/C/C- = 
2.0; D+/D/D- = 1.0; F = 0.0. 

The cumulative number of credits earned is scaled as a percentage of the total number of credits 
(core or subject-specific) required for graduation in a student's district. 



reading comprehension impacts of the two programs is not significant, and thus it cannot be 
concluded that one program was more effective than the other. Across all other student out- 
comes measured in this report (16 in total), 24 individual program impacts were statistically 
different from one another for only two outcomes (state test scores in science in the program 
year and social studies tests in the follow-up year). 



Poststudy Implementation of the ERO Programs 

After the two years of implementation required by the study, ERO schools and districts 
continued to receive SLC grant funding, but they were free to decide whether to continue the 



24 This includes eight types of outcome (GPA, credit accumulation, attendance, suspensions, and state test 
scores in each of the four core subject areas) measured in two follow-up years. 
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ERO programs or to use the funds to improve other aspects of their SLCs. Interviews were 
conducted with school-level staff to find out whether the study schools continued to use the 
ERO programs after the study-required implementation period, and if so, in what ways (if any) 
the programs have been modified to fit local circumstances. The study team was able to 
interview school-level staff from 30 of the 34 study schools. 

• Fourteen high schools (47 percent of the interviewed schools) continued to offer the 
ERO program after the end of the study-required implementation period. Schools that 
continued to use the ERO programs modified them. 

Deviations from the implementation conditions required by the study aimed at increas- 
ing the number of students served by the programs (for example, increasing class size [10 
schools], or serving students other than ninth-graders [seven schools]) and/or lengthening their 
duration [seven schools]). Also, eight schools modified the content of the programs and nine 
schools reduced the levels of professional development and technical assistance provided to 
teachers. The modifications that these schools made to the programs may alter the programs’ 
effectiveness relative to the impact findings. 
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Chapter 1 

Introduction 



According to the National Assessment of Educational Progress (NAEP), just over 70 
percent of students nationally arrive in high school with reading skills that are below “profi- 
cient” — defined as demonstrating competency over challenging subject matter. 1 Of these 
students, nearly half do not exhibit even partial mastery of the knowledge and skills that are 
fundamental to proficient work at grade level. 2 These limitations in literacy skills are a major 
source of course failure, high school dropout, and poor performance in postsecondary educa- 
tion. 3 While research is beginning to emerge about the special needs of staving adolescent 
readers, very little is known about effective interventions aimed at addressing these needs. 4 

To help fill this gap and to provide evidence-based guidance to practitioners, the U.S. 
Department of Education initiated the Enhanced Reading Opportunities (ERO) study — a 
demonstration and rigorous evaluation of supplemental literacy programs targeted to ninth- 
grade students with limited literacy skills. 5 The ERO demonstration involves 34 high schools 
from 10 school districts across the country that implemented one of two supplemental literacy 
programs: Reading Apprenticeship Academic Literacy (RAAL), designed by WestEd, or 
Xtreme Reading, designed by the University of Kansas Center for Research on Learning. The 
programs are supplemental in that they consist of a year-long course that replaces a ninth-grade 
elective class rather than a core academic class. They aim to help striving adolescent readers 
develop the strategies and routines used by proficient readers and to motivate them to read more 
and to apply these strategies to a wide range of texts. The districts that participated in the study 
were selected through a special grant competition organized by the U.S. Department of Educa- 
tion’s Office of Vocational and Adult Education (OVAE). 6 Anticipating that there would be 



Uhe NAEP is a nationally representative assessment of student achievement overseen by the National 
Center for Education Statistics (http://nces.ed.gov/nationsreportcard/). 

2 Lutkus, Rampey, and Donahue (2006) provide an analysis of NAEP reading results for urban school dis- 
tricts in the context of the national NAEP performance trends. 

3 Camevale (2001); Kamil (2003); Snow and Biancarosa (2003). 

4 Biancarosa and Snow (2004). 

5 The ERO study is known more formally as “An Evaluation of the Impact of Supplemental Literacy Inter- 
ventions in Freshman Academies.” 

6 For a complete application package for the special competition, see U.S. Department of Education 
(2005). The special grant competition was part of OVAE’s Smaller Learning Communities (SLC) initiative 
and was designed to provide extra funding to qualifying districts for the implementation of the supplemental 
literacy programs and participation in the ERO evaluation. The grants also included hinds for general support 
of the SLC initiatives under way in the districts. In 2006, responsibility for the SLC initiative and for the 
special ERO grants was moved from OVAE to Office of Elementary and Secondary Education (OESE). 
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first-year start-up challenges, the U.S. Department of Education supported a demonstration and 
evaluation of these programs that included two years of program operations in the study schools 
(2005-2006 and 2006-2007 school years), resulting in two cohorts of participating ninth-grade 
students. MDRC — a nonprofit, nonpartisan social policy research organization — conducted 
the evaluation in partnership with the American Institutes for Research (AIR) and Survey 
Research Management (SRM). 

The ERO study assesses the impact of the two supplemental literacy programs on stu- 
dents’ reading comprehension skills and on their general performance in high school. The first 
two reports for the study focused on measuring how well the programs were implemented and 
evaluated the programs’ impact on students’ reading achievement and behaviors. 7 This report — 
the final of three reports from this evaluation — examines the impact of the programs on 
students’ high school outcomes during the year in which they were enrolled in the ERO pro- 
grams (ninth grade), as well as the following school year (tenth grade for most students). The 
academic outcomes that are the focus of this report include students’ course grades, their accu- 
mulation of the credits they need to graduate, as well as their scores on state standardized 
assessments. The report also examines the programs’ effect on students’ school behaviors, which 
include attendance and disciplinary infractions. These school-based outcomes — course grades, 
credit earning, state assessment scores, attendance, and disciplinary infractions — represent 
measures typically used by schools and districts to assess how their students are doing. 8 In 
addition, as discussed more fully later in this chapter, analyzing the impacts on these outcomes 
tests the subsequent step in a theory of change. This theory posits that addressing the reading 
needs of students will help them do better not only on reading assessments, but also in their 
schoolwork more broadly. To set a foundation for the presentation and discussion of the analyses 
of the school-based outcomes, this report reviews the findings from the first two reports. Thus, 
this report provides comprehensive coverage of what has been learned from this evaluation. 

The remainder of this chapter provides background on the ERO demonstration, includ- 
ing descriptions of the two interventions selected for the project, their implementation in study 
schools in 2005-2006 and 2006-2007, and an overview of the research design used to assess 
impacts on key student outcomes. 



7 See Kemple et al. (2008) and Corrin et al. (2008). Chapter 3 is a review of the findings from these first 
two reports — the impact findings on reading test scores and reading behaviors as well as the implementation 
findings from research on the operation of the two programs. 

throughout the report there will be references to “high school outcomes” or “school-based outcomes,” 
which refer to outcomes from school records data — that is, outcomes from data regularly collected by the 
schools participating in the study. The reading achievement and reading behaviors outcomes that were the 
focus of the fust two reports are considered “study-based outcomes,” as they were measured using instruments 
administered specifically for this evaluation. 
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